Skip to content

feat(retrieval): EsCompatRetriever + xerjSidecar — Stage 2b mechanism - #12

Merged
thatswiftdev merged 2 commits into
mainfrom
wt/xerj-stage2
Oct 21, 2026
Merged

thatswiftdev merged 2 commits into
mainfrom
wt/xerj-stage2

Conversation

@thatswiftdev

Copy link
Copy Markdown
Contributor

What

Stage 2b of the xerj adoption arc (Lionidas plans/20261021T032458Z-xerj-stage2-plan.md, part B; evidence: Stage-1 spike on lionidas wt/xerj-stage1).

  • EsCompatRetriever — a Retriever over any Elasticsearch-compatible HTTP endpoint. BM25-only by design: the Stage-1 A/B measured parity with in-memory BM25 (0.400 vs 0.400 top-3) at ~42–51 ms vs ~18 s per query over a 26k-doc corpus, while hybrid/semantic regressed and cost 10–30 s per fresh query. Hit ids carry provenance (<docSha256>#<ordinal> / note:<id>), so consumers' citation contracts are untouched. Failures throw — degradation is the runner's decision.
  • xerjSidecar() — optional one-call runner: locate/spawn the sidecar (loopback, pinned flags per xerj's security model), health-wait, resumable reconcile (mget existence checks + bulk retries) from a consumer-supplied corpus enumerator (butler has no list-all seam). Returns {retriever, degraded, indexedCount, stop}; every failure is a degraded result with the explicit install hint — never a throw, never a silent install, nothing spawns at import.
  • Helpers chunkToEsDoc / noteToEsDoc; hermetic tests over a fake ES endpoint (injectable fetch/spawn).

Verification

  • pnpm test: 36 files, 209 passed, 5 skipped (pre-existing pg/env skips) — including 7 new tests covering provenance mapping, throw contract, attach/reconcile/idempotent-skip, and both degraded paths (attach-only miss; spawn-never-healthy kills the child and surfaces the install hint).
  • Genericity law held: no Lionidas vocabulary in butler; the corpus enumerator is the consumer's.

Notes for reviewer

  • Merge is operator-only (per repo workflow); lionidas pins this commit in its Stage-2 wiring PR after this merges.
  • BM25-only is deliberate (Stage-1 verdict); the embedder question reopens only with a multilingual model behind proxy mode.

BM25-only Retriever over any ES-compatible endpoint with provenance-carrying
ids; optional one-call sidecar runner (spawn/health/resumable reconcile from
a consumer-supplied corpus enumerator; degraded results, never throws, never
installs silently). Hermetic tests over a fake endpoint. v0.7.0.
@thatswiftdev
thatswiftdev merged commit 82f2130 into main Oct 21, 2026
1 check passed
@thatswiftdev
thatswiftdev deleted the wt/xerj-stage2 branch October 21, 2026 04:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant